All the Perl that's Practical to Extract and Report

Parallel Testing

Eric Wilhelm on 2007-09-11T09:03:49

So I decided to finally (after all of this compatibility work) take TAP::Harness out for a spin and make TAP::Harness::Parallel. That comes with some pain about the output formatting API (and I'll push it to CPAN as soon as we stabilize that -- right now you have to ride along and get TAP::Harness from its svn too (and possibly r491 because we're still working-through the single-process assumptions in the formatter.)) But, the preliminary results are between 40-60% reduced waiting. If your test suite takes more than 10s to run, it is probably worth it.

runtests -r -j 9

Note: I symlink bin/prove to runtests -- the Test::Harness transition isn't quite complete.

Expect about 40% time reduction in typical tests suites. The perl 5.8.8 core tests dropped to 200s from 500s. The parrot tests seem to be playing along as well. (Both of these require the directory-based partitioning.)

The biggest problem is resource conflicts. Have you ever run your tests simultaneously before? Probably not -- it is likely that they reuse tempfile names, server ports, etc. Time to start tacking $$ onto your tempfiles or using File::Temp. Of course, the directory-based partitioning (run each directory as a process) tends to help drop-in compatibility, but can block performance if things are unbalanced.

Next steps probably involve more switches and various forms of partitioning.

Thank you!

Ovid on 2007-09-11T09:09:59

Eric, this is a huge win for us. People used to ask "why are we rewriting Test::Harness?" It's because of things like this that they've stopped asking this question.

Thank you!

Adrian on 2007-09-11T09:48:59

I'd hacked evil equivalents several times in the past but never had the time and brains (or the new T::H :-) to make something sanely generic.

You are most exellent!

Eric is being diplomatic...

AndyArmstrong on 2007-09-11T12:28:17

...about the fact that it's my refactoring of our display code that's causing him a load of pain.

Great work in the face of considerable adversity Eric :)

Lesson of the day...

jk2addict on 2007-09-11T13:35:26

with Parallel...make sure to use unique tmp file names. :-)

Lesson #2: make sure no single test depends on a certain run order for lesson #1.

Is there a -shuffle or some other method in the new stuff to randomize the order tests are run in. I was addicted to that feature in Apache::Test. It keeps you honest.

The next question will be:

jk2addict on 2007-09-11T13:39:46

Is there, or will there be a way that test scripts can alert harness that they can't, or shouldn't be run in parallel due to some external resource issues?

Re:The next question will be:

Aristotle on 2007-09-11T16:36:44

That’s what partitioning is for. Within a partition, tests run sequentially, not in parallel.

Re: Sequential partitions

Eric Wilhelm on 2007-09-11T18:48:34

Yep. Most simply, partitioning based on dirname($test) will work for most people. The trick is how to add options to prove when '--jobs N' is triggering a run-time load of the optional module. Perhaps it requires its own frontend, but that's actually quite easy due to prove being abstracted into App::Prove (well, except for the lack of composite-ing support in Getopt::Long.)

Thanks Goodness

thepler on 2007-09-15T20:29:06

I have had to hack something like this into Test::Harness before, and it was annoying. I haven't looked at TAP::Harness yet, so hopefully it is better.

One thing I remember being particularly annoying... The harness shouldn't assume that it is running the test directly. The test may actually run on another host, and the stdout/stderr saved to a file on that host and then copied back and fed to the harness. I'm not saying ::Parallel needs to provide all that, but it should be fairly easy to do with a subclass or plugin or somesuch.

I'm looking forward to seeing how this works out. Hopefully I'll be able to change our test suite at $work to use it.